{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# 多层感知机\n",
    ":label:`sec_mlp`\n",
    "\n",
    "在 :numref:`chap_linear` 中，我们介绍了softmax回归（ :numref:`sec_softmax` ），然后我们从零开始实现softmax回归（ :numref:`sec_softmax_scratch` ），接着使用高级API实现了算法（ :numref:`sec_softmax_concise` ），并训练分类器从低分辨率图像中识别10类服装。在这个过程中，我们学习了如何处理数据，将输出转换为有效的概率分布，并应用适当的损失函数，根据模型参数最小化损失。我们已经在简单的线性模型背景下掌握了这些知识，现在我们可以开始对深度神经网络的探索，这也是本书主要涉及的一类模型。\n",
    "\n",
    "## 隐藏层\n",
    "\n",
    "我们在 :numref:`subsec_linear_model`中描述了仿射变换，它是一个带有偏置项的线性变换。首先，回想一下如 :numref:`fig_softmaxreg`中所示的softmax回归的模型结构。该模型通过单个仿射变换将我们的输入直接映射到输出，然后进行softmax操作。如果我们的标签通过仿射变换后确实与我们的输入数据相关，那么这种方法就足够了。但是，仿射变换中的*线性*是一个很强的假设。\n",
    "\n",
    "### 线性模型可能会出错\n",
    "\n",
    "例如，线性意味着*单调*假设：特征的任何增大都会导致模型输出增大（如果对应的权重为正），或者导致模型输出减少（如果对应的权重为负）。有时这是有道理的。例如，如果我们试图预测一个人是否会偿还贷款。我们可以认为，在其他条件不变的情况下，收入较高的申请人总是比收入较低的申请人更有可能偿还贷款。但是，虽然收入与还款概率存在单调性，但它们不是线性相关的。收入从0增加到5万，可能比从100万增加到105万带来更大的还款可能性。处理这一问题的一种方法是对我们的数据进行预处理，使线性变得更合理，如使用收入的对数作为我们的特征。\n",
    "\n",
    "我们可以很容易地找出违反单调性的例子。例如，我们想要根据体温预测死亡率。对于体温高于37摄氏度的人来说，温度越高风险越大。然而，对于体温低于37摄氏度的人来说，温度越高风险就越低。在这种情况下，我们也可以通过一些巧妙的预处理来解决问题。例如，我们可以使用与37摄氏度的距离作为特征。\n",
    "\n",
    "但是，如何对猫和狗的图像进行分类呢？增加位置(13, 17)处像素的强度是否总是增加(或总是降低)图像描绘狗的可能性？对线性模型的依赖对应于一个隐含的假设，即区分猫和狗的唯一要求是评估单个像素的强度。在一个倒置图像保留类别的世界里，这种方法注定会失败。\n",
    "\n",
    "与我们前面的例子相比，这里的线性很荒谬，而且我们难以通过简单的预处理来解决这个问题。这是因为任何像素的重要性都以复杂的方式取决于该像素的上下文（周围像素的值）。我们的数据可能会有一种表示，这种表示会考虑到我们的特征之间的相关交互作用。在此表示的基础上建立一个线性模型可能会是合适的，但我们不知道如何手动计算这么一种表示。对于深度神经网络，我们使用观测数据来联合学习隐藏层表示和应用于该表示的线性预测器。\n",
    "\n",
    "### 在网络中加入隐藏层\n",
    "\n",
    "我们可以通过在网络中加入一个或多个隐藏层来克服线性模型的限制，使其能处理更普遍的函数关系类型。要做到这一点，最简单的方法是将许多全连接层堆叠在一起。每一层都输出到上面的层，直到生成最后的输出。我们可以把前$L-1$层看作表示，把最后一层看作线性预测器。这种架构通常称为*多层感知机*（multilayer perceptron），通常缩写为*MLP*。下面，我们以图的方式描述了多层感知机（ :numref:`fig_mlp`）。\n",
    "\n",
    "![一个单隐藏层的多层感知机，具有5个隐藏单元](../img/mlp.svg)\n",
    ":label:`fig_mlp`\n",
    "\n",
    "这个多层感知机有4个输入，3个输出，其隐藏层包含5个隐藏单元。输入层不涉及任何计算，因此使用此网络产生输出只需要实现隐藏层和输出层的计算；因此，这个多层感知机中的层数为2。注意，这两个层都是全连接的。每个输入都会影响隐藏层中的每个神经元，而隐藏层中的每个神经元又会影响输出层中的每个神经元。\n",
    "\n",
    "然而，正如 :numref:`subsec_parameterization-cost-fc-layers` 所说，具有全连接层的多层感知机的参数开销可能会高得令人望而却步，\n",
    "即使在不改变输入或输出大小的情况下，也可能促使在参数节约和模型有效性之间进行权衡 :cite:`Zhang.Tay.Zhang.ea.2021`。\n",
    "\n",
    "### 从线性到非线性\n",
    "\n",
    "跟之前的章节一样，我们通过矩阵$\\mathbf{X} \\in \\mathbb{R}^{n \\times d}$来表示$n$个样本的小批量，其中每个样本具有$d$个输入(特征)。对于具有$h$个隐藏单元的单隐藏层多层感知机，用$\\mathbf{H} \\in \\mathbb{R}^{n \\times h}$表示隐藏层的输出，称为*隐藏表示*（hidden representations）。在数学或代码中，$\\mathbf{H}$也被称为*隐藏层变量*（hidden-layer variable）或*隐藏变量*（hidden variable）。\n",
    "因为隐藏层和输出层都是全连接的，所以我们具有隐藏层权重$\\mathbf{W}^{(1)} \\in \\mathbb{R}^{d \\times h}$和隐藏层偏置$\\mathbf{b}^{(1)} \\in \\mathbb{R}^{1 \\times h}$以及输出层权重$\\mathbf{W}^{(2)} \\in \\mathbb{R}^{h \\times q}$和输出层偏置$\\mathbf{b}^{(2)} \\in \\mathbb{R}^{1 \\times q}$。形式上，我们按如下方式计算单隐藏层多层感知机的输出$\\mathbf{O} \\in \\mathbb{R}^{n \\times q}$：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{H} & = \\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)}, \\\\\n",
    "    \\mathbf{O} & = \\mathbf{H}\\mathbf{W}^{(2)} + \\mathbf{b}^{(2)}.\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "注意，在添加隐藏层之后，模型现在需要跟踪和更新额外的参数。可我们能从中得到什么好处呢？你可能会惊讶地发现：在上面定义的模型里，我们没有好处！原因很简单。上面的隐藏单元由输入的仿射函数给出，而输出（softmax操作前）只是隐藏单元的仿射函数。仿射函数的仿射函数本身就是仿射函数。但是我们之前的线性模型已经能够表示任何仿射函数。\n",
    "\n",
    "我们可以证明这一等价性，即对于任意权重值，我们只需合并隐藏层，便可产生具有参数$\\mathbf{W} = \\mathbf{W}^{(1)}\\mathbf{W}^{(2)}$和$\\mathbf{b} = \\mathbf{b}^{(1)} \\mathbf{W}^{(2)} + \\mathbf{b}^{(2)}$的等价单层模型：\n",
    "\n",
    "$$\n",
    "\\mathbf{O} = (\\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)})\\mathbf{W}^{(2)} + \\mathbf{b}^{(2)} = \\mathbf{X} \\mathbf{W}^{(1)}\\mathbf{W}^{(2)} + \\mathbf{b}^{(1)} \\mathbf{W}^{(2)} + \\mathbf{b}^{(2)} = \\mathbf{X} \\mathbf{W} + \\mathbf{b}.\n",
    "$$\n",
    "\n",
    "为了发挥多层结构的潜力，我们还需要一个额外的关键要素：在仿射变换之后对每个隐藏单元应用非线性的*激活函数*（activation function）$\\sigma$。激活函数的输出（例如，$\\sigma(\\cdot)$）被称为*激活值*（activations）。一般来说，有了激活函数，就不可能再将我们的多层感知机退化成线性模型：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{H} & = \\sigma(\\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)}), \\\\\n",
    "    \\mathbf{O} & = \\mathbf{H}\\mathbf{W}^{(2)} + \\mathbf{b}^{(2)}.\\\\\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "由于$\\mathbf{X}$中的每一行对应于小批量中的一个样本，出于记号习惯的考量，我们定义非线性函数$\\sigma$也以按行的方式作用于其输入，即一次计算一个样本。我们在 :numref:`subsec_softmax_vectorization` 中以相同的方式使用了softmax符号来表示按行操作。但是在本节中，我们应用于隐藏层的激活函数通常不仅仅是按行的，而且也是按元素。\n",
    "这意味着在计算每一层的线性部分之后，我们可以计算每个激活值，而不需要查看其他隐藏单元所取的值。对于大多数激活函数都是这样。\n",
    "\n",
    "为了构建更通用的多层感知机，我们可以继续堆叠这样的隐藏层，例如，$\\mathbf{H}^{(1)} = \\sigma_1(\\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)})$和$\\mathbf{H}^{(2)} = \\sigma_2(\\mathbf{H}^{(1)} \\mathbf{W}^{(2)} + \\mathbf{b}^{(2)})$，一层叠一层，从而产生更有表达能力的模型。\n",
    "\n",
    "### 通用近似定理\n",
    "\n",
    "多层感知机可以通过隐藏神经元捕捉到我们输入之间复杂的相互作用，这些神经元依赖于每个输入的值。我们可以很容易地设计隐藏节点来执行任意计算。例如，在一对输入上进行基本逻辑操作。多层感知机是通用近似器。即使是网络只有一个隐藏层，给定足够的神经元（可能非常多）和正确的权重，我们可以对任意函数建模，尽管实际中学习该函数是很困难的。你可能认为神经网络有点像C语言。C语言和任何其他现代编程语言一样，能够表达任何可计算的程序。但实际上，想出一个符合规范的程序才是最困难的部分。\n",
    "\n",
    "而且，虽然一个单隐层网络能学习任何函数，但并不意味着应该尝试使用单隐藏层网络来解决所有问题。事实上，通过使用更深（而不是更广）的网络，我们可以更容易地逼近许多函数。我们将在后面的章节中进行更细致的讨论。\n",
    "\n",
    "## 激活函数\n",
    ":label:`subsec:activation-functions`\n",
    "\n",
    "激活函数通过计算加权和并加上偏置来确定神经元是否应该被激活。它们是将输入信号转换为输出的可微运算。大多数激活函数都是非线性的。由于激活函数是深度学习的基础，下面(**简要介绍一些常见的激活函数**)。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "origin_pos": 2,
    "scrolled": true,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import paddle\n",
    "from d2l import torch as d2l\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 4
   },
   "source": [
    "### ReLU函数\n",
    "\n",
    "最受欢迎的选择是*线性整流单元*（Rectified linear unit，*ReLU*），因为它实现简单，同时在各种预测任务中表现良好。\n",
    "[**ReLU提供了一种非常简单的非线性变换**]。\n",
    "给定元素$x$，ReLU函数被定义为该元素与$0$的最大值：\n",
    "\n",
    "（**$$\\operatorname{ReLU}(x) = \\max(x, 0).$$**）\n",
    "\n",
    "通俗地说，ReLU函数通过将相应的激活值设为0来仅保留正元素并丢弃所有负元素。为了直观感受一下，我们可以画出函数的曲线图。正如从图中所看到，激活函数是分段线性的。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\ipykernel\\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "import pd2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "origin_pos": 6,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\d2l\\torch.py:41: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n",
      "  display.set_matplotlib_formats('svg')\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 320.440625 180.65625\" width=\"320.440625pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-10T15:02:34.563318</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 320.440625 180.65625 \r\n",
       "L 320.440625 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 34.240625 143.1 \r\n",
       "L 313.240625 143.1 \r\n",
       "L 313.240625 7.2 \r\n",
       "L 34.240625 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 46.922443 143.1 \r\n",
       "L 46.922443 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m0e67982ed3\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"46.922443\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −8 -->\r\n",
       "      <g transform=\"translate(39.551349 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 78.826409 143.1 \r\n",
       "L 78.826409 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"78.826409\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −6 -->\r\n",
       "      <g transform=\"translate(71.455315 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 110.730375 143.1 \r\n",
       "L 110.730375 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"110.730375\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <g transform=\"translate(103.359281 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 142.634341 143.1 \r\n",
       "L 142.634341 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"142.634341\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(135.263247 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 174.538307 143.1 \r\n",
       "L 174.538307 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"174.538307\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(171.357057 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 206.442273 143.1 \r\n",
       "L 206.442273 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.442273\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(203.261023 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 238.346239 143.1 \r\n",
       "L 238.346239 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"238.346239\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(235.164989 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_8\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 270.250205 143.1 \r\n",
       "L 270.250205 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"270.250205\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(267.068955 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_9\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 302.154171 143.1 \r\n",
       "L 302.154171 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"302.154171\" xlink:href=\"#m0e67982ed3\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(298.972921 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- x -->\r\n",
       "     <g transform=\"translate(170.78125 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 34.240625 136.922727 \r\n",
       "L 313.240625 136.922727 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"me9efe7f96a\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"34.240625\" xlink:href=\"#me9efe7f96a\" y=\"136.922727\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 34.240625 105.645356 \r\n",
       "L 313.240625 105.645356 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"34.240625\" xlink:href=\"#me9efe7f96a\" y=\"105.645356\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(20.878125 109.444575)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 34.240625 74.367984 \r\n",
       "L 313.240625 74.367984 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"34.240625\" xlink:href=\"#me9efe7f96a\" y=\"74.367984\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(20.878125 78.167203)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 34.240625 43.090613 \r\n",
       "L 313.240625 43.090613 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"34.240625\" xlink:href=\"#me9efe7f96a\" y=\"43.090613\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_14\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(20.878125 46.889832)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <path clip-path=\"url(#p4508c42ddb)\" d=\"M 34.240625 11.813242 \r\n",
       "L 313.240625 11.813242 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"34.240625\" xlink:href=\"#me9efe7f96a\" y=\"11.813242\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_15\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(20.878125 15.61246)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_16\">\r\n",
       "     <!-- relu(x) -->\r\n",
       "     <g transform=\"translate(14.798438 91.588281)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 544 1381 \r\n",
       "L 544 3500 \r\n",
       "L 1119 3500 \r\n",
       "L 1119 1403 \r\n",
       "Q 1119 906 1312 657 \r\n",
       "Q 1506 409 1894 409 \r\n",
       "Q 2359 409 2629 706 \r\n",
       "Q 2900 1003 2900 1516 \r\n",
       "L 2900 3500 \r\n",
       "L 3475 3500 \r\n",
       "L 3475 0 \r\n",
       "L 2900 0 \r\n",
       "L 2900 538 \r\n",
       "Q 2691 219 2414 64 \r\n",
       "Q 2138 -91 1772 -91 \r\n",
       "Q 1169 -91 856 284 \r\n",
       "Q 544 659 544 1381 \r\n",
       "z\r\n",
       "M 1991 3584 \r\n",
       "L 1991 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-75\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1984 4856 \r\n",
       "Q 1566 4138 1362 3434 \r\n",
       "Q 1159 2731 1159 2009 \r\n",
       "Q 1159 1288 1364 580 \r\n",
       "Q 1569 -128 1984 -844 \r\n",
       "L 1484 -844 \r\n",
       "Q 1016 -109 783 600 \r\n",
       "Q 550 1309 550 2009 \r\n",
       "Q 550 2706 781 3412 \r\n",
       "Q 1013 4119 1484 4856 \r\n",
       "L 1984 4856 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 513 4856 \r\n",
       "L 1013 4856 \r\n",
       "Q 1481 4119 1714 3412 \r\n",
       "Q 1947 2706 1947 2009 \r\n",
       "Q 1947 1309 1714 600 \r\n",
       "Q 1481 -109 1013 -844 \r\n",
       "L 513 -844 \r\n",
       "Q 928 -128 1133 580 \r\n",
       "Q 1338 1288 1338 2009 \r\n",
       "Q 1338 2731 1133 3434 \r\n",
       "Q 928 4138 513 4856 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"38.863281\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"100.386719\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"128.169922\" xlink:href=\"#DejaVuSans-75\"/>\r\n",
       "      <use x=\"191.548828\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"230.5625\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"289.742188\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_29\">\r\n",
       "    <path clip-path=\"url(#p4508c42ddb)\" d=\"M 46.922443 136.922727 \r\n",
       "L 174.538221 136.922727 \r\n",
       "L 300.558807 13.377273 \r\n",
       "L 300.558807 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 34.240625 143.1 \r\n",
       "L 34.240625 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 313.240625 143.1 \r\n",
       "L 313.240625 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 34.240625 143.1 \r\n",
       "L 313.240625 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 34.240625 7.2 \r\n",
       "L 313.240625 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p4508c42ddb\">\r\n",
       "   <rect height=\"135.9\" width=\"279\" x=\"34.240625\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 360x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = paddle.arange(-8.0, 8.0, 0.1, dtype='float32')\n",
    "x.stop_gradient = False\n",
    "y = paddle.nn.functional.relu(x)\n",
    "\n",
    "d2l.plot(x.detach().numpy(), y.detach().numpy(), 'x', 'relu(x)', figsize=(5, 2.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 8
   },
   "source": [
    "当输入为负时，ReLU函数的导数为0，而当输入为正时，ReLU函数的导数为1。注意，当输入值精确等于0时，ReLU函数不可导。在此时，我们默认使用左侧的导数，即当输入为0时导数为0。我们可以忽略这种情况，因为输入可能永远都不会是0。这里用上一句古老的谚语，“如果微妙的边界条件很重要，我们很可能是在研究数学而非工程”，这个观点正好适用于这里。下面我们绘制ReLU函数的导数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "origin_pos": 10,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\ipykernel\\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\paddle\\fluid\\dygraph\\varbase_patch_methods.py:382: UserWarning: \n",
      "Warning:\n",
      "tensor.grad will return the tensor value of the gradient. This is an incompatible upgrade for tensor.grad API.  It's return type changes from numpy.ndarray in version 2.0 to paddle.Tensor in version 2.1.0.  If you want to get the numpy value of the gradient, you can use :code:`x.grad.numpy()` \n",
      "  warnings.warn(warning_msg)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\d2l\\torch.py:41: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n",
      "  display.set_matplotlib_formats('svg')\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 329.98125 180.65625\" width=\"329.98125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-10T15:02:36.313136</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 329.98125 180.65625 \r\n",
       "L 329.98125 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 322.78125 143.1 \r\n",
       "L 322.78125 7.2 \r\n",
       "L 43.78125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 56.463068 143.1 \r\n",
       "L 56.463068 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"ma068312cf9\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"56.463068\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −8 -->\r\n",
       "      <g transform=\"translate(49.091974 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 88.367034 143.1 \r\n",
       "L 88.367034 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"88.367034\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −6 -->\r\n",
       "      <g transform=\"translate(80.99594 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 120.271 143.1 \r\n",
       "L 120.271 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"120.271\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <g transform=\"translate(112.899906 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 152.174966 143.1 \r\n",
       "L 152.174966 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"152.174966\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(144.803872 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 184.078932 143.1 \r\n",
       "L 184.078932 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"184.078932\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(180.897682 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 215.982898 143.1 \r\n",
       "L 215.982898 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"215.982898\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(212.801648 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 247.886864 143.1 \r\n",
       "L 247.886864 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"247.886864\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(244.705614 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_8\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 279.79083 143.1 \r\n",
       "L 279.79083 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"279.79083\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(276.60958 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_9\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 311.694796 143.1 \r\n",
       "L 311.694796 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"311.694796\" xlink:href=\"#ma068312cf9\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(308.513546 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- x -->\r\n",
       "     <g transform=\"translate(180.321875 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 43.78125 136.922727 \r\n",
       "L 322.78125 136.922727 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m7227803a93\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7227803a93\" y=\"136.922727\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 684 794 \r\n",
       "L 1344 794 \r\n",
       "L 1344 0 \r\n",
       "L 684 0 \r\n",
       "L 684 794 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 43.78125 112.213636 \r\n",
       "L 322.78125 112.213636 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7227803a93\" y=\"112.213636\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- 0.2 -->\r\n",
       "      <g transform=\"translate(20.878125 116.012855)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 43.78125 87.504545 \r\n",
       "L 322.78125 87.504545 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7227803a93\" y=\"87.504545\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- 0.4 -->\r\n",
       "      <g transform=\"translate(20.878125 91.303764)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 43.78125 62.795455 \r\n",
       "L 322.78125 62.795455 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7227803a93\" y=\"62.795455\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_14\">\r\n",
       "      <!-- 0.6 -->\r\n",
       "      <g transform=\"translate(20.878125 66.594673)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 43.78125 38.086364 \r\n",
       "L 322.78125 38.086364 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7227803a93\" y=\"38.086364\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_15\">\r\n",
       "      <!-- 0.8 -->\r\n",
       "      <g transform=\"translate(20.878125 41.885582)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <path clip-path=\"url(#pf392be81ab)\" d=\"M 43.78125 13.377273 \r\n",
       "L 322.78125 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7227803a93\" y=\"13.377273\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_16\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_17\">\r\n",
       "     <!-- grad of relu -->\r\n",
       "     <g transform=\"translate(14.798438 104.192969)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2906 1791 \r\n",
       "Q 2906 2416 2648 2759 \r\n",
       "Q 2391 3103 1925 3103 \r\n",
       "Q 1463 3103 1205 2759 \r\n",
       "Q 947 2416 947 1791 \r\n",
       "Q 947 1169 1205 825 \r\n",
       "Q 1463 481 1925 481 \r\n",
       "Q 2391 481 2648 825 \r\n",
       "Q 2906 1169 2906 1791 \r\n",
       "z\r\n",
       "M 3481 434 \r\n",
       "Q 3481 -459 3084 -895 \r\n",
       "Q 2688 -1331 1869 -1331 \r\n",
       "Q 1566 -1331 1297 -1286 \r\n",
       "Q 1028 -1241 775 -1147 \r\n",
       "L 775 -588 \r\n",
       "Q 1028 -725 1275 -790 \r\n",
       "Q 1522 -856 1778 -856 \r\n",
       "Q 2344 -856 2625 -561 \r\n",
       "Q 2906 -266 2906 331 \r\n",
       "L 2906 616 \r\n",
       "Q 2728 306 2450 153 \r\n",
       "Q 2172 0 1784 0 \r\n",
       "Q 1141 0 747 490 \r\n",
       "Q 353 981 353 1791 \r\n",
       "Q 353 2603 747 3093 \r\n",
       "Q 1141 3584 1784 3584 \r\n",
       "Q 2172 3584 2450 3431 \r\n",
       "Q 2728 3278 2906 2969 \r\n",
       "L 2906 3500 \r\n",
       "L 3481 3500 \r\n",
       "L 3481 434 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-67\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2906 2969 \r\n",
       "L 2906 4863 \r\n",
       "L 3481 4863 \r\n",
       "L 3481 0 \r\n",
       "L 2906 0 \r\n",
       "L 2906 525 \r\n",
       "Q 2725 213 2448 61 \r\n",
       "Q 2172 -91 1784 -91 \r\n",
       "Q 1150 -91 751 415 \r\n",
       "Q 353 922 353 1747 \r\n",
       "Q 353 2572 751 3078 \r\n",
       "Q 1150 3584 1784 3584 \r\n",
       "Q 2172 3584 2448 3432 \r\n",
       "Q 2725 3281 2906 2969 \r\n",
       "z\r\n",
       "M 947 1747 \r\n",
       "Q 947 1113 1208 752 \r\n",
       "Q 1469 391 1925 391 \r\n",
       "Q 2381 391 2643 752 \r\n",
       "Q 2906 1113 2906 1747 \r\n",
       "Q 2906 2381 2643 2742 \r\n",
       "Q 2381 3103 1925 3103 \r\n",
       "Q 1469 3103 1208 2742 \r\n",
       "Q 947 2381 947 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-64\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2375 4863 \r\n",
       "L 2375 4384 \r\n",
       "L 1825 4384 \r\n",
       "Q 1516 4384 1395 4259 \r\n",
       "Q 1275 4134 1275 3809 \r\n",
       "L 1275 3500 \r\n",
       "L 2222 3500 \r\n",
       "L 2222 3053 \r\n",
       "L 1275 3053 \r\n",
       "L 1275 0 \r\n",
       "L 697 0 \r\n",
       "L 697 3053 \r\n",
       "L 147 3053 \r\n",
       "L 147 3500 \r\n",
       "L 697 3500 \r\n",
       "L 697 3744 \r\n",
       "Q 697 4328 969 4595 \r\n",
       "Q 1241 4863 1831 4863 \r\n",
       "L 2375 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 544 1381 \r\n",
       "L 544 3500 \r\n",
       "L 1119 3500 \r\n",
       "L 1119 1403 \r\n",
       "Q 1119 906 1312 657 \r\n",
       "Q 1506 409 1894 409 \r\n",
       "Q 2359 409 2629 706 \r\n",
       "Q 2900 1003 2900 1516 \r\n",
       "L 2900 3500 \r\n",
       "L 3475 3500 \r\n",
       "L 3475 0 \r\n",
       "L 2900 0 \r\n",
       "L 2900 538 \r\n",
       "Q 2691 219 2414 64 \r\n",
       "Q 2138 -91 1772 -91 \r\n",
       "Q 1169 -91 856 284 \r\n",
       "Q 544 659 544 1381 \r\n",
       "z\r\n",
       "M 1991 3584 \r\n",
       "L 1991 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-75\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-67\"/>\r\n",
       "      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"104.589844\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"165.869141\" xlink:href=\"#DejaVuSans-64\"/>\r\n",
       "      <use x=\"229.345703\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "      <use x=\"261.132812\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"322.314453\" xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"357.519531\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "      <use x=\"389.306641\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"428.169922\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"489.693359\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"517.476562\" xlink:href=\"#DejaVuSans-75\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_31\">\r\n",
       "    <path clip-path=\"url(#pf392be81ab)\" d=\"M 56.463068 136.922727 \r\n",
       "L 184.078846 136.922727 \r\n",
       "L 185.674045 13.377273 \r\n",
       "L 310.099432 13.377273 \r\n",
       "L 310.099432 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 43.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 322.78125 143.1 \r\n",
       "L 322.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 322.78125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 43.78125 7.2 \r\n",
       "L 322.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pf392be81ab\">\r\n",
       "   <rect height=\"135.9\" width=\"279\" x=\"43.78125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 360x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y.backward(paddle.ones_like(x), retain_graph=True)\n",
    "d2l.plot(x.detach().numpy(), x.grad.numpy(), 'x', 'grad of relu', figsize=(5, 2.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "使用ReLU的原因是，它求导表现得特别好：要么让参数消失，要么让参数通过。这使得优化表现得更好，并且ReLU减轻了困扰以往神经网络的梯度消失问题（稍后将详细介绍）。\n",
    "\n",
    "注意，ReLU函数有许多变体，包括*参数化ReLU*（Parameterized ReLU，*pReLU*）函数 :cite:`He.Zhang.Ren.ea.2015`。该变体为ReLU添加了一个线性项，因此即使参数是负的，某些信息仍然可以通过：\n",
    "\n",
    "$$\\operatorname{pReLU}(x) = \\max(0, x) + \\alpha \\min(0, x).$$\n",
    "\n",
    "### sigmoid函数\n",
    "\n",
    "[**对于一个定义域在$\\mathbb{R}$中的输入，*sigmoid函数*将输入变换为区间(0, 1)上的输出**]。因此，sigmoid通常称为*挤压函数*（squashing function）：它将范围(-inf, inf)中的任意输入压缩到区间(0, 1)中的某个值：\n",
    "\n",
    "（**$$\\operatorname{sigmoid}(x) = \\frac{1}{1 + \\exp(-x)}.$$**）\n",
    "\n",
    "在最早的神经网络中，科学家们感兴趣的是对“激发”或“不激发”的生物神经元进行建模。因此，这一领域的先驱，如人工神经元的发明者麦卡洛克和皮茨，从他们开始就专注于阈值单元。阈值单元在其输入低于某个阈值时取值0，当输入超过阈值时取值1。\n",
    "\n",
    "当人们的注意力逐渐转移到基于梯度的学习时，sigmoid函数是一个自然的选择，因为它是一个平滑的、可微的阈值单元近似。当我们想要将输出视作二分类问题的概率时，sigmoid仍然被广泛用作输出单元上的激活函数（你可以将sigmoid视为softmax的特例）。然而，sigmoid在隐藏层中已经较少使用，它在大部分时候已经被更简单、更容易训练的ReLU所取代。在后面关于循环神经网络的章节中，我们将描述利用sigmoid单元来控制时序信息流动的结构。\n",
    "\n",
    "下面，我们绘制sigmoid函数。注意，当输入接近0时，sigmoid函数接近线性变换。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "origin_pos": 14,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\ipykernel\\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\d2l\\torch.py:41: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n",
      "  display.set_matplotlib_formats('svg')\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 329.98125 180.65625\" width=\"329.98125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-10T15:02:37.731723</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 329.98125 180.65625 \r\n",
       "L 329.98125 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 322.78125 143.1 \r\n",
       "L 322.78125 7.2 \r\n",
       "L 43.78125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 56.463068 143.1 \r\n",
       "L 56.463068 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"mc79e0f9aa6\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"56.463068\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −8 -->\r\n",
       "      <g transform=\"translate(49.091974 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 88.367034 143.1 \r\n",
       "L 88.367034 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"88.367034\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −6 -->\r\n",
       "      <g transform=\"translate(80.99594 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 120.271 143.1 \r\n",
       "L 120.271 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"120.271\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <g transform=\"translate(112.899906 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 152.174966 143.1 \r\n",
       "L 152.174966 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"152.174966\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(144.803872 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 184.078932 143.1 \r\n",
       "L 184.078932 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"184.078932\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(180.897682 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 215.982898 143.1 \r\n",
       "L 215.982898 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"215.982898\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(212.801648 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 247.886864 143.1 \r\n",
       "L 247.886864 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"247.886864\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(244.705614 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_8\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 279.79083 143.1 \r\n",
       "L 279.79083 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"279.79083\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(276.60958 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_9\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 311.694796 143.1 \r\n",
       "L 311.694796 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"311.694796\" xlink:href=\"#mc79e0f9aa6\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(308.513546 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- x -->\r\n",
       "     <g transform=\"translate(180.321875 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 43.78125 136.964188 \r\n",
       "L 322.78125 136.964188 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m4f0e828363\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m4f0e828363\" y=\"136.964188\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <g transform=\"translate(20.878125 140.763406)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 684 794 \r\n",
       "L 1344 794 \r\n",
       "L 1344 0 \r\n",
       "L 684 0 \r\n",
       "L 684 794 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 43.78125 112.23764 \r\n",
       "L 322.78125 112.23764 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m4f0e828363\" y=\"112.23764\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- 0.2 -->\r\n",
       "      <g transform=\"translate(20.878125 116.036859)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 43.78125 87.511093 \r\n",
       "L 322.78125 87.511093 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m4f0e828363\" y=\"87.511093\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- 0.4 -->\r\n",
       "      <g transform=\"translate(20.878125 91.310312)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 43.78125 62.784546 \r\n",
       "L 322.78125 62.784546 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m4f0e828363\" y=\"62.784546\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_14\">\r\n",
       "      <!-- 0.6 -->\r\n",
       "      <g transform=\"translate(20.878125 66.583765)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 43.78125 38.057999 \r\n",
       "L 322.78125 38.057999 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m4f0e828363\" y=\"38.057999\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_15\">\r\n",
       "      <!-- 0.8 -->\r\n",
       "      <g transform=\"translate(20.878125 41.857218)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 43.78125 13.331452 \r\n",
       "L 322.78125 13.331452 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m4f0e828363\" y=\"13.331452\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_16\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <g transform=\"translate(20.878125 17.130671)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_17\">\r\n",
       "     <!-- sigmoid(x) -->\r\n",
       "     <g transform=\"translate(14.798438 101.671875)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2834 3397 \r\n",
       "L 2834 2853 \r\n",
       "Q 2591 2978 2328 3040 \r\n",
       "Q 2066 3103 1784 3103 \r\n",
       "Q 1356 3103 1142 2972 \r\n",
       "Q 928 2841 928 2578 \r\n",
       "Q 928 2378 1081 2264 \r\n",
       "Q 1234 2150 1697 2047 \r\n",
       "L 1894 2003 \r\n",
       "Q 2506 1872 2764 1633 \r\n",
       "Q 3022 1394 3022 966 \r\n",
       "Q 3022 478 2636 193 \r\n",
       "Q 2250 -91 1575 -91 \r\n",
       "Q 1294 -91 989 -36 \r\n",
       "Q 684 19 347 128 \r\n",
       "L 347 722 \r\n",
       "Q 666 556 975 473 \r\n",
       "Q 1284 391 1588 391 \r\n",
       "Q 1994 391 2212 530 \r\n",
       "Q 2431 669 2431 922 \r\n",
       "Q 2431 1156 2273 1281 \r\n",
       "Q 2116 1406 1581 1522 \r\n",
       "L 1381 1569 \r\n",
       "Q 847 1681 609 1914 \r\n",
       "Q 372 2147 372 2553 \r\n",
       "Q 372 3047 722 3315 \r\n",
       "Q 1072 3584 1716 3584 \r\n",
       "Q 2034 3584 2315 3537 \r\n",
       "Q 2597 3491 2834 3397 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2906 1791 \r\n",
       "Q 2906 2416 2648 2759 \r\n",
       "Q 2391 3103 1925 3103 \r\n",
       "Q 1463 3103 1205 2759 \r\n",
       "Q 947 2416 947 1791 \r\n",
       "Q 947 1169 1205 825 \r\n",
       "Q 1463 481 1925 481 \r\n",
       "Q 2391 481 2648 825 \r\n",
       "Q 2906 1169 2906 1791 \r\n",
       "z\r\n",
       "M 3481 434 \r\n",
       "Q 3481 -459 3084 -895 \r\n",
       "Q 2688 -1331 1869 -1331 \r\n",
       "Q 1566 -1331 1297 -1286 \r\n",
       "Q 1028 -1241 775 -1147 \r\n",
       "L 775 -588 \r\n",
       "Q 1028 -725 1275 -790 \r\n",
       "Q 1522 -856 1778 -856 \r\n",
       "Q 2344 -856 2625 -561 \r\n",
       "Q 2906 -266 2906 331 \r\n",
       "L 2906 616 \r\n",
       "Q 2728 306 2450 153 \r\n",
       "Q 2172 0 1784 0 \r\n",
       "Q 1141 0 747 490 \r\n",
       "Q 353 981 353 1791 \r\n",
       "Q 353 2603 747 3093 \r\n",
       "Q 1141 3584 1784 3584 \r\n",
       "Q 2172 3584 2450 3431 \r\n",
       "Q 2728 3278 2906 2969 \r\n",
       "L 2906 3500 \r\n",
       "L 3481 3500 \r\n",
       "L 3481 434 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-67\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3328 2828 \r\n",
       "Q 3544 3216 3844 3400 \r\n",
       "Q 4144 3584 4550 3584 \r\n",
       "Q 5097 3584 5394 3201 \r\n",
       "Q 5691 2819 5691 2113 \r\n",
       "L 5691 0 \r\n",
       "L 5113 0 \r\n",
       "L 5113 2094 \r\n",
       "Q 5113 2597 4934 2840 \r\n",
       "Q 4756 3084 4391 3084 \r\n",
       "Q 3944 3084 3684 2787 \r\n",
       "Q 3425 2491 3425 1978 \r\n",
       "L 3425 0 \r\n",
       "L 2847 0 \r\n",
       "L 2847 2094 \r\n",
       "Q 2847 2600 2669 2842 \r\n",
       "Q 2491 3084 2119 3084 \r\n",
       "Q 1678 3084 1418 2786 \r\n",
       "Q 1159 2488 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1356 3278 1631 3431 \r\n",
       "Q 1906 3584 2284 3584 \r\n",
       "Q 2666 3584 2933 3390 \r\n",
       "Q 3200 3197 3328 2828 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6d\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2906 2969 \r\n",
       "L 2906 4863 \r\n",
       "L 3481 4863 \r\n",
       "L 3481 0 \r\n",
       "L 2906 0 \r\n",
       "L 2906 525 \r\n",
       "Q 2725 213 2448 61 \r\n",
       "Q 2172 -91 1784 -91 \r\n",
       "Q 1150 -91 751 415 \r\n",
       "Q 353 922 353 1747 \r\n",
       "Q 353 2572 751 3078 \r\n",
       "Q 1150 3584 1784 3584 \r\n",
       "Q 2172 3584 2448 3432 \r\n",
       "Q 2725 3281 2906 2969 \r\n",
       "z\r\n",
       "M 947 1747 \r\n",
       "Q 947 1113 1208 752 \r\n",
       "Q 1469 391 1925 391 \r\n",
       "Q 2381 391 2643 752 \r\n",
       "Q 2906 1113 2906 1747 \r\n",
       "Q 2906 2381 2643 2742 \r\n",
       "Q 2381 3103 1925 3103 \r\n",
       "Q 1469 3103 1208 2742 \r\n",
       "Q 947 2381 947 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-64\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1984 4856 \r\n",
       "Q 1566 4138 1362 3434 \r\n",
       "Q 1159 2731 1159 2009 \r\n",
       "Q 1159 1288 1364 580 \r\n",
       "Q 1569 -128 1984 -844 \r\n",
       "L 1484 -844 \r\n",
       "Q 1016 -109 783 600 \r\n",
       "Q 550 1309 550 2009 \r\n",
       "Q 550 2706 781 3412 \r\n",
       "Q 1013 4119 1484 4856 \r\n",
       "L 1984 4856 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 513 4856 \r\n",
       "L 1013 4856 \r\n",
       "Q 1481 4119 1714 3412 \r\n",
       "Q 1947 2706 1947 2009 \r\n",
       "Q 1947 1309 1714 600 \r\n",
       "Q 1481 -109 1013 -844 \r\n",
       "L 513 -844 \r\n",
       "Q 928 -128 1133 580 \r\n",
       "Q 1338 1288 1338 2009 \r\n",
       "Q 1338 2731 1133 3434 \r\n",
       "Q 928 4138 513 4856 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"52.099609\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"79.882812\" xlink:href=\"#DejaVuSans-67\"/>\r\n",
       "      <use x=\"143.359375\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "      <use x=\"240.771484\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"301.953125\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"329.736328\" xlink:href=\"#DejaVuSans-64\"/>\r\n",
       "      <use x=\"393.212891\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"432.226562\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"491.40625\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_31\">\r\n",
       "    <path clip-path=\"url(#p3f7f4ad29d)\" d=\"M 56.463068 136.922727 \r\n",
       "L 83.581413 136.737577 \r\n",
       "L 96.342988 136.460986 \r\n",
       "L 105.914168 136.050354 \r\n",
       "L 113.890152 135.464721 \r\n",
       "L 120.270939 134.740511 \r\n",
       "L 125.05653 133.98131 \r\n",
       "L 129.84212 132.971428 \r\n",
       "L 133.032513 132.122044 \r\n",
       "L 136.222907 131.100824 \r\n",
       "L 139.413301 129.877076 \r\n",
       "L 142.603694 128.416457 \r\n",
       "L 145.794088 126.68137 \r\n",
       "L 148.984481 124.631823 \r\n",
       "L 152.174875 122.226878 \r\n",
       "L 155.365272 119.426837 \r\n",
       "L 158.55567 116.196257 \r\n",
       "L 161.746067 112.50775 \r\n",
       "L 164.936464 108.346395 \r\n",
       "L 168.126862 103.714359 \r\n",
       "L 171.317259 98.635029 \r\n",
       "L 174.507656 93.155863 \r\n",
       "L 179.293251 84.351516 \r\n",
       "L 193.650036 57.140082 \r\n",
       "L 196.840433 51.660901 \r\n",
       "L 200.03083 46.581546 \r\n",
       "L 203.221227 41.949486 \r\n",
       "L 206.411624 37.788097 \r\n",
       "L 209.602022 34.099575 \r\n",
       "L 212.792419 30.868968 \r\n",
       "L 215.982816 28.068893 \r\n",
       "L 219.173212 25.663933 \r\n",
       "L 222.363605 23.614362 \r\n",
       "L 225.553999 21.879266 \r\n",
       "L 228.744392 20.41864 \r\n",
       "L 231.934786 19.194878 \r\n",
       "L 235.125179 18.173641 \r\n",
       "L 239.91077 16.955428 \r\n",
       "L 244.69636 16.036709 \r\n",
       "L 249.481954 15.346992 \r\n",
       "L 255.862741 14.68981 \r\n",
       "L 263.838725 14.158904 \r\n",
       "L 273.409906 13.786942 \r\n",
       "L 286.17148 13.536533 \r\n",
       "L 306.909038 13.38742 \r\n",
       "L 310.099432 13.377273 \r\n",
       "L 310.099432 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 43.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 322.78125 143.1 \r\n",
       "L 322.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 322.78125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 43.78125 7.2 \r\n",
       "L 322.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p3f7f4ad29d\">\r\n",
       "   <rect height=\"135.9\" width=\"279\" x=\"43.78125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 360x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = paddle.nn.functional.sigmoid(x)\n",
    "d2l.plot(x.detach().numpy(), y.detach().numpy(), 'x', 'sigmoid(x)', figsize=(5, 2.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 16
   },
   "source": [
    "sigmoid函数的导数为下面的公式：\n",
    "\n",
    "$$\\frac{d}{dx} \\operatorname{sigmoid}(x) = \\frac{\\exp(-x)}{(1 + \\exp(-x))^2} = \\operatorname{sigmoid}(x)\\left(1-\\operatorname{sigmoid}(x)\\right).$$\n",
    "\n",
    "sigmoid函数的导数图像如下所示。注意，当输入为0时，sigmoid函数的导数达到最大值0.25。而输入在任一方向上越远离0点，导数越接近0。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "origin_pos": 18,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\ipykernel\\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\paddle\\fluid\\dygraph\\varbase_patch_methods.py:382: UserWarning: \n",
      "Warning:\n",
      "tensor.grad will return the tensor value of the gradient. This is an incompatible upgrade for tensor.grad API.  It's return type changes from numpy.ndarray in version 2.0 to paddle.Tensor in version 2.1.0.  If you want to get the numpy value of the gradient, you can use :code:`x.grad.numpy()` \n",
      "  warnings.warn(warning_msg)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\d2l\\torch.py:41: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n",
      "  display.set_matplotlib_formats('svg')\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 336.34375 180.65625\" width=\"336.34375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-10T15:02:38.985726</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 336.34375 180.65625 \r\n",
       "L 336.34375 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 50.14375 143.1 \r\n",
       "L 329.14375 143.1 \r\n",
       "L 329.14375 7.2 \r\n",
       "L 50.14375 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 62.825568 143.1 \r\n",
       "L 62.825568 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m0106447935\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"62.825568\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −8 -->\r\n",
       "      <g transform=\"translate(55.454474 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 94.729534 143.1 \r\n",
       "L 94.729534 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.729534\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −6 -->\r\n",
       "      <g transform=\"translate(87.35844 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 126.6335 143.1 \r\n",
       "L 126.6335 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"126.6335\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <g transform=\"translate(119.262406 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 158.537466 143.1 \r\n",
       "L 158.537466 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"158.537466\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(151.166372 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 190.441432 143.1 \r\n",
       "L 190.441432 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"190.441432\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(187.260182 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 222.345398 143.1 \r\n",
       "L 222.345398 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"222.345398\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(219.164148 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 254.249364 143.1 \r\n",
       "L 254.249364 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"254.249364\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(251.068114 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_8\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 286.15333 143.1 \r\n",
       "L 286.15333 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"286.15333\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(282.97208 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_9\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 318.057296 143.1 \r\n",
       "L 318.057296 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"318.057296\" xlink:href=\"#m0106447935\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(314.876046 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- x -->\r\n",
       "     <g transform=\"translate(186.684375 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 50.14375 137.088618 \r\n",
       "L 329.14375 137.088618 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m9b8a934a3f\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m9b8a934a3f\" y=\"137.088618\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 0.00 -->\r\n",
       "      <g transform=\"translate(20.878125 140.887837)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 684 794 \r\n",
       "L 1344 794 \r\n",
       "L 1344 0 \r\n",
       "L 684 0 \r\n",
       "L 684 794 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 50.14375 112.346349 \r\n",
       "L 329.14375 112.346349 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m9b8a934a3f\" y=\"112.346349\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- 0.05 -->\r\n",
       "      <g transform=\"translate(20.878125 116.145568)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 691 4666 \r\n",
       "L 3169 4666 \r\n",
       "L 3169 4134 \r\n",
       "L 1269 4134 \r\n",
       "L 1269 2991 \r\n",
       "Q 1406 3038 1543 3061 \r\n",
       "Q 1681 3084 1819 3084 \r\n",
       "Q 2600 3084 3056 2656 \r\n",
       "Q 3513 2228 3513 1497 \r\n",
       "Q 3513 744 3044 326 \r\n",
       "Q 2575 -91 1722 -91 \r\n",
       "Q 1428 -91 1123 -41 \r\n",
       "Q 819 9 494 109 \r\n",
       "L 494 744 \r\n",
       "Q 775 591 1075 516 \r\n",
       "Q 1375 441 1709 441 \r\n",
       "Q 2250 441 2565 725 \r\n",
       "Q 2881 1009 2881 1497 \r\n",
       "Q 2881 1984 2565 2268 \r\n",
       "Q 2250 2553 1709 2553 \r\n",
       "Q 1456 2553 1204 2497 \r\n",
       "Q 953 2441 691 2322 \r\n",
       "L 691 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 50.14375 87.60408 \r\n",
       "L 329.14375 87.60408 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m9b8a934a3f\" y=\"87.60408\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- 0.10 -->\r\n",
       "      <g transform=\"translate(20.878125 91.403299)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 50.14375 62.861811 \r\n",
       "L 329.14375 62.861811 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m9b8a934a3f\" y=\"62.861811\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_14\">\r\n",
       "      <!-- 0.15 -->\r\n",
       "      <g transform=\"translate(20.878125 66.66103)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 50.14375 38.119542 \r\n",
       "L 329.14375 38.119542 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m9b8a934a3f\" y=\"38.119542\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_15\">\r\n",
       "      <!-- 0.20 -->\r\n",
       "      <g transform=\"translate(20.878125 41.918761)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <path clip-path=\"url(#p245e56335c)\" d=\"M 50.14375 13.377273 \r\n",
       "L 329.14375 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m9b8a934a3f\" y=\"13.377273\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_16\">\r\n",
       "      <!-- 0.25 -->\r\n",
       "      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_17\">\r\n",
       "     <!-- grad of sigmoid -->\r\n",
       "     <g transform=\"translate(14.798438 114.276562)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2906 1791 \r\n",
       "Q 2906 2416 2648 2759 \r\n",
       "Q 2391 3103 1925 3103 \r\n",
       "Q 1463 3103 1205 2759 \r\n",
       "Q 947 2416 947 1791 \r\n",
       "Q 947 1169 1205 825 \r\n",
       "Q 1463 481 1925 481 \r\n",
       "Q 2391 481 2648 825 \r\n",
       "Q 2906 1169 2906 1791 \r\n",
       "z\r\n",
       "M 3481 434 \r\n",
       "Q 3481 -459 3084 -895 \r\n",
       "Q 2688 -1331 1869 -1331 \r\n",
       "Q 1566 -1331 1297 -1286 \r\n",
       "Q 1028 -1241 775 -1147 \r\n",
       "L 775 -588 \r\n",
       "Q 1028 -725 1275 -790 \r\n",
       "Q 1522 -856 1778 -856 \r\n",
       "Q 2344 -856 2625 -561 \r\n",
       "Q 2906 -266 2906 331 \r\n",
       "L 2906 616 \r\n",
       "Q 2728 306 2450 153 \r\n",
       "Q 2172 0 1784 0 \r\n",
       "Q 1141 0 747 490 \r\n",
       "Q 353 981 353 1791 \r\n",
       "Q 353 2603 747 3093 \r\n",
       "Q 1141 3584 1784 3584 \r\n",
       "Q 2172 3584 2450 3431 \r\n",
       "Q 2728 3278 2906 2969 \r\n",
       "L 2906 3500 \r\n",
       "L 3481 3500 \r\n",
       "L 3481 434 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-67\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2906 2969 \r\n",
       "L 2906 4863 \r\n",
       "L 3481 4863 \r\n",
       "L 3481 0 \r\n",
       "L 2906 0 \r\n",
       "L 2906 525 \r\n",
       "Q 2725 213 2448 61 \r\n",
       "Q 2172 -91 1784 -91 \r\n",
       "Q 1150 -91 751 415 \r\n",
       "Q 353 922 353 1747 \r\n",
       "Q 353 2572 751 3078 \r\n",
       "Q 1150 3584 1784 3584 \r\n",
       "Q 2172 3584 2448 3432 \r\n",
       "Q 2725 3281 2906 2969 \r\n",
       "z\r\n",
       "M 947 1747 \r\n",
       "Q 947 1113 1208 752 \r\n",
       "Q 1469 391 1925 391 \r\n",
       "Q 2381 391 2643 752 \r\n",
       "Q 2906 1113 2906 1747 \r\n",
       "Q 2906 2381 2643 2742 \r\n",
       "Q 2381 3103 1925 3103 \r\n",
       "Q 1469 3103 1208 2742 \r\n",
       "Q 947 2381 947 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-64\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2375 4863 \r\n",
       "L 2375 4384 \r\n",
       "L 1825 4384 \r\n",
       "Q 1516 4384 1395 4259 \r\n",
       "Q 1275 4134 1275 3809 \r\n",
       "L 1275 3500 \r\n",
       "L 2222 3500 \r\n",
       "L 2222 3053 \r\n",
       "L 1275 3053 \r\n",
       "L 1275 0 \r\n",
       "L 697 0 \r\n",
       "L 697 3053 \r\n",
       "L 147 3053 \r\n",
       "L 147 3500 \r\n",
       "L 697 3500 \r\n",
       "L 697 3744 \r\n",
       "Q 697 4328 969 4595 \r\n",
       "Q 1241 4863 1831 4863 \r\n",
       "L 2375 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2834 3397 \r\n",
       "L 2834 2853 \r\n",
       "Q 2591 2978 2328 3040 \r\n",
       "Q 2066 3103 1784 3103 \r\n",
       "Q 1356 3103 1142 2972 \r\n",
       "Q 928 2841 928 2578 \r\n",
       "Q 928 2378 1081 2264 \r\n",
       "Q 1234 2150 1697 2047 \r\n",
       "L 1894 2003 \r\n",
       "Q 2506 1872 2764 1633 \r\n",
       "Q 3022 1394 3022 966 \r\n",
       "Q 3022 478 2636 193 \r\n",
       "Q 2250 -91 1575 -91 \r\n",
       "Q 1294 -91 989 -36 \r\n",
       "Q 684 19 347 128 \r\n",
       "L 347 722 \r\n",
       "Q 666 556 975 473 \r\n",
       "Q 1284 391 1588 391 \r\n",
       "Q 1994 391 2212 530 \r\n",
       "Q 2431 669 2431 922 \r\n",
       "Q 2431 1156 2273 1281 \r\n",
       "Q 2116 1406 1581 1522 \r\n",
       "L 1381 1569 \r\n",
       "Q 847 1681 609 1914 \r\n",
       "Q 372 2147 372 2553 \r\n",
       "Q 372 3047 722 3315 \r\n",
       "Q 1072 3584 1716 3584 \r\n",
       "Q 2034 3584 2315 3537 \r\n",
       "Q 2597 3491 2834 3397 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3328 2828 \r\n",
       "Q 3544 3216 3844 3400 \r\n",
       "Q 4144 3584 4550 3584 \r\n",
       "Q 5097 3584 5394 3201 \r\n",
       "Q 5691 2819 5691 2113 \r\n",
       "L 5691 0 \r\n",
       "L 5113 0 \r\n",
       "L 5113 2094 \r\n",
       "Q 5113 2597 4934 2840 \r\n",
       "Q 4756 3084 4391 3084 \r\n",
       "Q 3944 3084 3684 2787 \r\n",
       "Q 3425 2491 3425 1978 \r\n",
       "L 3425 0 \r\n",
       "L 2847 0 \r\n",
       "L 2847 2094 \r\n",
       "Q 2847 2600 2669 2842 \r\n",
       "Q 2491 3084 2119 3084 \r\n",
       "Q 1678 3084 1418 2786 \r\n",
       "Q 1159 2488 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1356 3278 1631 3431 \r\n",
       "Q 1906 3584 2284 3584 \r\n",
       "Q 2666 3584 2933 3390 \r\n",
       "Q 3200 3197 3328 2828 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6d\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-67\"/>\r\n",
       "      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"104.589844\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"165.869141\" xlink:href=\"#DejaVuSans-64\"/>\r\n",
       "      <use x=\"229.345703\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "      <use x=\"261.132812\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"322.314453\" xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"357.519531\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "      <use x=\"389.306641\" xlink:href=\"#DejaVuSans-73\"/>\r\n",
       "      <use x=\"441.40625\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"469.189453\" xlink:href=\"#DejaVuSans-67\"/>\r\n",
       "      <use x=\"532.666016\" xlink:href=\"#DejaVuSans-6d\"/>\r\n",
       "      <use x=\"630.078125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"691.259766\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"719.042969\" xlink:href=\"#DejaVuSans-64\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_31\">\r\n",
       "    <path clip-path=\"url(#p245e56335c)\" d=\"M 62.825568 136.922727 \r\n",
       "L 78.777536 136.6382 \r\n",
       "L 88.348717 136.269131 \r\n",
       "L 96.324701 135.740416 \r\n",
       "L 102.705488 135.082731 \r\n",
       "L 107.491078 134.38865 \r\n",
       "L 112.276668 133.457994 \r\n",
       "L 115.467062 132.668643 \r\n",
       "L 118.657455 131.711532 \r\n",
       "L 121.847849 130.552867 \r\n",
       "L 125.038242 129.152946 \r\n",
       "L 128.228636 127.465561 \r\n",
       "L 131.41903 125.437574 \r\n",
       "L 134.609423 123.008828 \r\n",
       "L 137.799817 120.112583 \r\n",
       "L 140.99021 116.676817 \r\n",
       "L 144.180604 112.626753 \r\n",
       "L 147.370997 107.889054 \r\n",
       "L 150.561391 102.398275 \r\n",
       "L 153.751784 96.105919 \r\n",
       "L 156.942178 88.992479 \r\n",
       "L 160.132573 81.082123 \r\n",
       "L 163.322971 72.45922 \r\n",
       "L 168.108567 58.564396 \r\n",
       "L 174.489362 39.796358 \r\n",
       "L 177.679759 31.236619 \r\n",
       "L 179.274958 27.375226 \r\n",
       "L 180.870156 23.875981 \r\n",
       "L 182.465355 20.798273 \r\n",
       "L 184.060553 18.196828 \r\n",
       "L 185.655751 16.119659 \r\n",
       "L 187.25095 14.606252 \r\n",
       "L 188.846148 13.686072 \r\n",
       "L 190.441346 13.377273 \r\n",
       "L 192.036545 13.686005 \r\n",
       "L 193.631743 14.606119 \r\n",
       "L 195.226941 16.119453 \r\n",
       "L 196.822139 18.196578 \r\n",
       "L 198.417338 20.797963 \r\n",
       "L 200.012536 23.87562 \r\n",
       "L 201.607734 27.374821 \r\n",
       "L 203.202933 31.23617 \r\n",
       "L 206.39333 39.795871 \r\n",
       "L 211.178926 53.806707 \r\n",
       "L 215.964522 67.926895 \r\n",
       "L 219.154919 76.851182 \r\n",
       "L 222.345316 85.132852 \r\n",
       "L 225.535712 92.651164 \r\n",
       "L 228.726105 99.354043 \r\n",
       "L 231.916499 105.241062 \r\n",
       "L 235.106892 110.347952 \r\n",
       "L 238.297286 114.733026 \r\n",
       "L 241.487679 118.466624 \r\n",
       "L 244.678073 121.623395 \r\n",
       "L 247.868467 124.277174 \r\n",
       "L 251.05886 126.497638 \r\n",
       "L 254.249254 128.34824 \r\n",
       "L 257.439651 129.885714 \r\n",
       "L 260.630045 131.159634 \r\n",
       "L 263.820438 132.212952 \r\n",
       "L 267.010832 133.082326 \r\n",
       "L 271.796422 134.108084 \r\n",
       "L 276.582012 134.873661 \r\n",
       "L 282.962799 135.599466 \r\n",
       "L 289.343587 136.088434 \r\n",
       "L 297.31957 136.480971 \r\n",
       "L 308.485948 136.786478 \r\n",
       "L 316.461932 136.905286 \r\n",
       "L 316.461932 136.905286 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 50.14375 143.1 \r\n",
       "L 50.14375 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 329.14375 143.1 \r\n",
       "L 329.14375 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 50.14375 143.1 \r\n",
       "L 329.14375 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 50.14375 7.2 \r\n",
       "L 329.14375 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p245e56335c\">\r\n",
       "   <rect height=\"135.9\" width=\"279\" x=\"50.14375\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 360x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 清除以前的梯度。\n",
    "x.clear_gradient()\n",
    "y.backward(paddle.ones_like(x),retain_graph=True)\n",
    "d2l.plot(x.detach().numpy(), x.grad.numpy(), 'x', 'grad of sigmoid', figsize=(5, 2.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 20
   },
   "source": [
    "### tanh函数\n",
    "\n",
    "与sigmoid函数类似，[**tanh(双曲正切)函数也能将其输入压缩转换到区间(-1, 1)上**]。tanh函数的公式如下：\n",
    "\n",
    "（**$$\\operatorname{tanh}(x) = \\frac{1 - \\exp(-2x)}{1 + \\exp(-2x)}.$$**）\n",
    "\n",
    "下面我们绘制tanh函数。注意，当输入在0附近时，tanh函数接近线性变换。函数的形状类似于sigmoid函数，不同的是tanh函数关于坐标系原点中心对称。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "origin_pos": 22,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\ipykernel\\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\d2l\\torch.py:41: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n",
      "  display.set_matplotlib_formats('svg')\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 338.360937 180.65625\" width=\"338.360937pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-10T15:02:40.200681</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 338.360937 180.65625 \r\n",
       "L 338.360937 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 52.160938 143.1 \r\n",
       "L 331.160937 143.1 \r\n",
       "L 331.160937 7.2 \r\n",
       "L 52.160938 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 64.842756 143.1 \r\n",
       "L 64.842756 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m29a929a39c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"64.842756\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −8 -->\r\n",
       "      <g transform=\"translate(57.471662 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 96.746722 143.1 \r\n",
       "L 96.746722 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"96.746722\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −6 -->\r\n",
       "      <g transform=\"translate(89.375628 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 128.650688 143.1 \r\n",
       "L 128.650688 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"128.650688\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <g transform=\"translate(121.279594 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 160.554654 143.1 \r\n",
       "L 160.554654 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"160.554654\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(153.18356 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 192.45862 143.1 \r\n",
       "L 192.45862 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.45862\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(189.27737 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 224.362586 143.1 \r\n",
       "L 224.362586 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"224.362586\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(221.181336 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 256.266551 143.1 \r\n",
       "L 256.266551 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"256.266551\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(253.085301 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_8\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 288.170517 143.1 \r\n",
       "L 288.170517 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"288.170517\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(284.989267 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_9\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 320.074483 143.1 \r\n",
       "L 320.074483 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"320.074483\" xlink:href=\"#m29a929a39c\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(316.893233 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- x -->\r\n",
       "     <g transform=\"translate(188.701562 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 52.160938 136.922727 \r\n",
       "L 331.160937 136.922727 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"m91cf890249\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m91cf890249\" y=\"136.922727\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- −1.0 -->\r\n",
       "      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 684 794 \r\n",
       "L 1344 794 \r\n",
       "L 1344 0 \r\n",
       "L 684 0 \r\n",
       "L 684 794 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 52.160938 106.03636 \r\n",
       "L 331.160937 106.03636 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m91cf890249\" y=\"106.03636\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- −0.5 -->\r\n",
       "      <g transform=\"translate(20.878125 109.835579)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 691 4666 \r\n",
       "L 3169 4666 \r\n",
       "L 3169 4134 \r\n",
       "L 1269 4134 \r\n",
       "L 1269 2991 \r\n",
       "Q 1406 3038 1543 3061 \r\n",
       "Q 1681 3084 1819 3084 \r\n",
       "Q 2600 3084 3056 2656 \r\n",
       "Q 3513 2228 3513 1497 \r\n",
       "Q 3513 744 3044 326 \r\n",
       "Q 2575 -91 1722 -91 \r\n",
       "Q 1428 -91 1123 -41 \r\n",
       "Q 819 9 494 109 \r\n",
       "L 494 744 \r\n",
       "Q 775 591 1075 516 \r\n",
       "Q 1375 441 1709 441 \r\n",
       "Q 2250 441 2565 725 \r\n",
       "Q 2881 1009 2881 1497 \r\n",
       "Q 2881 1984 2565 2268 \r\n",
       "Q 2250 2553 1709 2553 \r\n",
       "Q 1456 2553 1204 2497 \r\n",
       "Q 953 2441 691 2322 \r\n",
       "L 691 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"179.199219\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 52.160938 75.149993 \r\n",
       "L 331.160937 75.149993 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m91cf890249\" y=\"75.149993\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <g transform=\"translate(29.257812 78.949211)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 52.160938 44.263625 \r\n",
       "L 331.160937 44.263625 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m91cf890249\" y=\"44.263625\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_14\">\r\n",
       "      <!-- 0.5 -->\r\n",
       "      <g transform=\"translate(29.257812 48.062844)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <path clip-path=\"url(#pd13e1a1518)\" d=\"M 52.160938 13.377258 \r\n",
       "L 331.160937 13.377258 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"52.160938\" xlink:href=\"#m91cf890249\" y=\"13.377258\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_15\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <g transform=\"translate(29.257812 17.176477)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_16\">\r\n",
       "     <!-- tanh(x) -->\r\n",
       "     <g transform=\"translate(14.798437 93.372656)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1984 4856 \r\n",
       "Q 1566 4138 1362 3434 \r\n",
       "Q 1159 2731 1159 2009 \r\n",
       "Q 1159 1288 1364 580 \r\n",
       "Q 1569 -128 1984 -844 \r\n",
       "L 1484 -844 \r\n",
       "Q 1016 -109 783 600 \r\n",
       "Q 550 1309 550 2009 \r\n",
       "Q 550 2706 781 3412 \r\n",
       "Q 1013 4119 1484 4856 \r\n",
       "L 1984 4856 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-28\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 513 4856 \r\n",
       "L 1013 4856 \r\n",
       "Q 1481 4119 1714 3412 \r\n",
       "Q 1947 2706 1947 2009 \r\n",
       "Q 1947 1309 1714 600 \r\n",
       "Q 1481 -109 1013 -844 \r\n",
       "L 513 -844 \r\n",
       "Q 928 -128 1133 580 \r\n",
       "Q 1338 1288 1338 2009 \r\n",
       "Q 1338 2731 1133 3434 \r\n",
       "Q 928 4138 513 4856 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-29\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"100.488281\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "      <use x=\"163.867188\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "      <use x=\"227.246094\" xlink:href=\"#DejaVuSans-28\"/>\r\n",
       "      <use x=\"266.259766\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"325.439453\" xlink:href=\"#DejaVuSans-29\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_29\">\r\n",
       "    <path clip-path=\"url(#pd13e1a1518)\" d=\"M 64.842756 136.922727 \r\n",
       "L 135.031414 136.830557 \r\n",
       "L 144.602595 136.617248 \r\n",
       "L 150.983382 136.244922 \r\n",
       "L 155.768972 135.693235 \r\n",
       "L 158.959366 135.097482 \r\n",
       "L 162.149761 134.219423 \r\n",
       "L 163.74496 133.63683 \r\n",
       "L 165.340158 132.932809 \r\n",
       "L 166.935357 132.084033 \r\n",
       "L 168.530556 131.063539 \r\n",
       "L 170.125754 129.840659 \r\n",
       "L 171.720953 128.381075 \r\n",
       "L 173.316152 126.647219 \r\n",
       "L 174.91135 124.599135 \r\n",
       "L 176.506549 122.195891 \r\n",
       "L 178.101748 119.397836 \r\n",
       "L 179.696946 116.16955 \r\n",
       "L 181.292145 112.483657 \r\n",
       "L 182.887344 108.32525 \r\n",
       "L 184.482542 103.696493 \r\n",
       "L 186.077741 98.620761 \r\n",
       "L 189.268137 87.342724 \r\n",
       "L 197.244129 57.15512 \r\n",
       "L 198.839327 51.679791 \r\n",
       "L 200.434525 46.604013 \r\n",
       "L 202.029723 41.97521 \r\n",
       "L 203.624922 37.816751 \r\n",
       "L 205.220121 34.130803 \r\n",
       "L 206.815319 30.902473 \r\n",
       "L 208.410518 28.104374 \r\n",
       "L 210.005716 25.701097 \r\n",
       "L 211.600914 23.652969 \r\n",
       "L 213.196113 21.919076 \r\n",
       "L 214.791312 20.459473 \r\n",
       "L 216.38651 19.236568 \r\n",
       "L 217.981709 18.216055 \r\n",
       "L 219.576908 17.367261 \r\n",
       "L 222.767305 16.080621 \r\n",
       "L 225.957702 15.202548 \r\n",
       "L 229.148096 14.606773 \r\n",
       "L 233.933686 14.05507 \r\n",
       "L 238.719277 13.75018 \r\n",
       "L 246.695261 13.514709 \r\n",
       "L 261.052035 13.400001 \r\n",
       "L 312.098332 13.377295 \r\n",
       "L 318.479119 13.377273 \r\n",
       "L 318.479119 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 52.160938 143.1 \r\n",
       "L 52.160938 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 331.160937 143.1 \r\n",
       "L 331.160937 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 52.160938 143.1 \r\n",
       "L 331.160938 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 52.160938 7.2 \r\n",
       "L 331.160938 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pd13e1a1518\">\r\n",
       "   <rect height=\"135.9\" width=\"279\" x=\"52.160938\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 360x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = paddle.tanh(x)\n",
    "d2l.plot(x.detach().numpy(), y.detach().numpy(), 'x', 'tanh(x)', figsize=(5, 2.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 24
   },
   "source": [
    "tanh函数的导数是：\n",
    "\n",
    "$$\\frac{d}{dx} \\operatorname{tanh}(x) = 1 - \\operatorname{tanh}^2(x).$$\n",
    "\n",
    "tanh函数的导数图像如下所示。当输入接近0时，tanh函数的导数接近最大值1。与我们在sigmoid函数图像中看到的类似，输入在任一方向上越远离0点，导数越接近0。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "origin_pos": 26,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\ipykernel\\ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\paddle\\fluid\\dygraph\\varbase_patch_methods.py:382: UserWarning: \n",
      "Warning:\n",
      "tensor.grad will return the tensor value of the gradient. This is an incompatible upgrade for tensor.grad API.  It's return type changes from numpy.ndarray in version 2.0 to paddle.Tensor in version 2.1.0.  If you want to get the numpy value of the gradient, you can use :code:`x.grad.numpy()` \n",
      "  warnings.warn(warning_msg)\n",
      "C:\\Users\\WeiWu-GU\\anaconda3\\envs\\pte\\lib\\site-packages\\d2l\\torch.py:41: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n",
      "  display.set_matplotlib_formats('svg')\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 329.98125 180.65625\" width=\"329.98125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2021-09-10T15:02:41.156383</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 329.98125 180.65625 \r\n",
       "L 329.98125 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 322.78125 143.1 \r\n",
       "L 322.78125 7.2 \r\n",
       "L 43.78125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 56.463068 143.1 \r\n",
       "L 56.463068 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"mb62abecae7\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"56.463068\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- −8 -->\r\n",
       "      <g transform=\"translate(49.091974 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 678 2272 \r\n",
       "L 4684 2272 \r\n",
       "L 4684 1741 \r\n",
       "L 678 1741 \r\n",
       "L 678 2272 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 2216 \r\n",
       "Q 1584 2216 1326 1975 \r\n",
       "Q 1069 1734 1069 1313 \r\n",
       "Q 1069 891 1326 650 \r\n",
       "Q 1584 409 2034 409 \r\n",
       "Q 2484 409 2743 651 \r\n",
       "Q 3003 894 3003 1313 \r\n",
       "Q 3003 1734 2745 1975 \r\n",
       "Q 2488 2216 2034 2216 \r\n",
       "z\r\n",
       "M 1403 2484 \r\n",
       "Q 997 2584 770 2862 \r\n",
       "Q 544 3141 544 3541 \r\n",
       "Q 544 4100 942 4425 \r\n",
       "Q 1341 4750 2034 4750 \r\n",
       "Q 2731 4750 3128 4425 \r\n",
       "Q 3525 4100 3525 3541 \r\n",
       "Q 3525 3141 3298 2862 \r\n",
       "Q 3072 2584 2669 2484 \r\n",
       "Q 3125 2378 3379 2068 \r\n",
       "Q 3634 1759 3634 1313 \r\n",
       "Q 3634 634 3220 271 \r\n",
       "Q 2806 -91 2034 -91 \r\n",
       "Q 1263 -91 848 271 \r\n",
       "Q 434 634 434 1313 \r\n",
       "Q 434 1759 690 2068 \r\n",
       "Q 947 2378 1403 2484 \r\n",
       "z\r\n",
       "M 1172 3481 \r\n",
       "Q 1172 3119 1398 2916 \r\n",
       "Q 1625 2713 2034 2713 \r\n",
       "Q 2441 2713 2670 2916 \r\n",
       "Q 2900 3119 2900 3481 \r\n",
       "Q 2900 3844 2670 4047 \r\n",
       "Q 2441 4250 2034 4250 \r\n",
       "Q 1625 4250 1398 4047 \r\n",
       "Q 1172 3844 1172 3481 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 88.367034 143.1 \r\n",
       "L 88.367034 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"88.367034\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- −6 -->\r\n",
       "      <g transform=\"translate(80.99594 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2113 2584 \r\n",
       "Q 1688 2584 1439 2293 \r\n",
       "Q 1191 2003 1191 1497 \r\n",
       "Q 1191 994 1439 701 \r\n",
       "Q 1688 409 2113 409 \r\n",
       "Q 2538 409 2786 701 \r\n",
       "Q 3034 994 3034 1497 \r\n",
       "Q 3034 2003 2786 2293 \r\n",
       "Q 2538 2584 2113 2584 \r\n",
       "z\r\n",
       "M 3366 4563 \r\n",
       "L 3366 3988 \r\n",
       "Q 3128 4100 2886 4159 \r\n",
       "Q 2644 4219 2406 4219 \r\n",
       "Q 1781 4219 1451 3797 \r\n",
       "Q 1122 3375 1075 2522 \r\n",
       "Q 1259 2794 1537 2939 \r\n",
       "Q 1816 3084 2150 3084 \r\n",
       "Q 2853 3084 3261 2657 \r\n",
       "Q 3669 2231 3669 1497 \r\n",
       "Q 3669 778 3244 343 \r\n",
       "Q 2819 -91 2113 -91 \r\n",
       "Q 1303 -91 875 529 \r\n",
       "Q 447 1150 447 2328 \r\n",
       "Q 447 3434 972 4092 \r\n",
       "Q 1497 4750 2381 4750 \r\n",
       "Q 2619 4750 2861 4703 \r\n",
       "Q 3103 4656 3366 4563 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 120.271 143.1 \r\n",
       "L 120.271 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"120.271\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- −4 -->\r\n",
       "      <g transform=\"translate(112.899906 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 152.174966 143.1 \r\n",
       "L 152.174966 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"152.174966\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- −2 -->\r\n",
       "      <g transform=\"translate(144.803872 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\r\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 184.078932 143.1 \r\n",
       "L 184.078932 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"184.078932\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 0 -->\r\n",
       "      <g transform=\"translate(180.897682 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_6\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 215.982898 143.1 \r\n",
       "L 215.982898 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"215.982898\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_6\">\r\n",
       "      <!-- 2 -->\r\n",
       "      <g transform=\"translate(212.801648 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_7\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 247.886864 143.1 \r\n",
       "L 247.886864 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"247.886864\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 4 -->\r\n",
       "      <g transform=\"translate(244.705614 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_8\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 279.79083 143.1 \r\n",
       "L 279.79083 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"279.79083\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 6 -->\r\n",
       "      <g transform=\"translate(276.60958 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_9\">\r\n",
       "     <g id=\"line2d_17\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 311.694796 143.1 \r\n",
       "L 311.694796 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_18\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"311.694796\" xlink:href=\"#mb62abecae7\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 8 -->\r\n",
       "      <g transform=\"translate(308.513546 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- x -->\r\n",
       "     <g transform=\"translate(180.321875 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_19\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 43.78125 136.922727 \r\n",
       "L 322.78125 136.922727 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_20\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mc1173ddb84\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#mc1173ddb84\" y=\"136.922727\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_11\">\r\n",
       "      <!-- 0.0 -->\r\n",
       "      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 684 794 \r\n",
       "L 1344 794 \r\n",
       "L 1344 0 \r\n",
       "L 684 0 \r\n",
       "L 684 794 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_21\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 43.78125 112.213636 \r\n",
       "L 322.78125 112.213636 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_22\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#mc1173ddb84\" y=\"112.213636\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_12\">\r\n",
       "      <!-- 0.2 -->\r\n",
       "      <g transform=\"translate(20.878125 116.012855)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_23\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 43.78125 87.504545 \r\n",
       "L 322.78125 87.504545 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_24\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#mc1173ddb84\" y=\"87.504545\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_13\">\r\n",
       "      <!-- 0.4 -->\r\n",
       "      <g transform=\"translate(20.878125 91.303764)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_4\">\r\n",
       "     <g id=\"line2d_25\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 43.78125 62.795455 \r\n",
       "L 322.78125 62.795455 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_26\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#mc1173ddb84\" y=\"62.795455\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_14\">\r\n",
       "      <!-- 0.6 -->\r\n",
       "      <g transform=\"translate(20.878125 66.594673)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-36\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_5\">\r\n",
       "     <g id=\"line2d_27\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 43.78125 38.086364 \r\n",
       "L 322.78125 38.086364 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_28\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#mc1173ddb84\" y=\"38.086364\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_15\">\r\n",
       "      <!-- 0.8 -->\r\n",
       "      <g transform=\"translate(20.878125 41.885582)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-38\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_6\">\r\n",
       "     <g id=\"line2d_29\">\r\n",
       "      <path clip-path=\"url(#pd419019c3e)\" d=\"M 43.78125 13.377273 \r\n",
       "L 322.78125 13.377273 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_30\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#mc1173ddb84\" y=\"13.377273\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_16\">\r\n",
       "      <!-- 1.0 -->\r\n",
       "      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\r\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_17\">\r\n",
       "     <!-- grad of tanh -->\r\n",
       "     <g transform=\"translate(14.798438 105.977344)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2906 1791 \r\n",
       "Q 2906 2416 2648 2759 \r\n",
       "Q 2391 3103 1925 3103 \r\n",
       "Q 1463 3103 1205 2759 \r\n",
       "Q 947 2416 947 1791 \r\n",
       "Q 947 1169 1205 825 \r\n",
       "Q 1463 481 1925 481 \r\n",
       "Q 2391 481 2648 825 \r\n",
       "Q 2906 1169 2906 1791 \r\n",
       "z\r\n",
       "M 3481 434 \r\n",
       "Q 3481 -459 3084 -895 \r\n",
       "Q 2688 -1331 1869 -1331 \r\n",
       "Q 1566 -1331 1297 -1286 \r\n",
       "Q 1028 -1241 775 -1147 \r\n",
       "L 775 -588 \r\n",
       "Q 1028 -725 1275 -790 \r\n",
       "Q 1522 -856 1778 -856 \r\n",
       "Q 2344 -856 2625 -561 \r\n",
       "Q 2906 -266 2906 331 \r\n",
       "L 2906 616 \r\n",
       "Q 2728 306 2450 153 \r\n",
       "Q 2172 0 1784 0 \r\n",
       "Q 1141 0 747 490 \r\n",
       "Q 353 981 353 1791 \r\n",
       "Q 353 2603 747 3093 \r\n",
       "Q 1141 3584 1784 3584 \r\n",
       "Q 2172 3584 2450 3431 \r\n",
       "Q 2728 3278 2906 2969 \r\n",
       "L 2906 3500 \r\n",
       "L 3481 3500 \r\n",
       "L 3481 434 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-67\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2906 2969 \r\n",
       "L 2906 4863 \r\n",
       "L 3481 4863 \r\n",
       "L 3481 0 \r\n",
       "L 2906 0 \r\n",
       "L 2906 525 \r\n",
       "Q 2725 213 2448 61 \r\n",
       "Q 2172 -91 1784 -91 \r\n",
       "Q 1150 -91 751 415 \r\n",
       "Q 353 922 353 1747 \r\n",
       "Q 353 2572 751 3078 \r\n",
       "Q 1150 3584 1784 3584 \r\n",
       "Q 2172 3584 2448 3432 \r\n",
       "Q 2725 3281 2906 2969 \r\n",
       "z\r\n",
       "M 947 1747 \r\n",
       "Q 947 1113 1208 752 \r\n",
       "Q 1469 391 1925 391 \r\n",
       "Q 2381 391 2643 752 \r\n",
       "Q 2906 1113 2906 1747 \r\n",
       "Q 2906 2381 2643 2742 \r\n",
       "Q 2381 3103 1925 3103 \r\n",
       "Q 1469 3103 1208 2742 \r\n",
       "Q 947 2381 947 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-64\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2375 4863 \r\n",
       "L 2375 4384 \r\n",
       "L 1825 4384 \r\n",
       "Q 1516 4384 1395 4259 \r\n",
       "Q 1275 4134 1275 3809 \r\n",
       "L 1275 3500 \r\n",
       "L 2222 3500 \r\n",
       "L 2222 3053 \r\n",
       "L 1275 3053 \r\n",
       "L 1275 0 \r\n",
       "L 697 0 \r\n",
       "L 697 3053 \r\n",
       "L 147 3053 \r\n",
       "L 147 3500 \r\n",
       "L 697 3500 \r\n",
       "L 697 3744 \r\n",
       "Q 697 4328 969 4595 \r\n",
       "Q 1241 4863 1831 4863 \r\n",
       "L 2375 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-66\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-67\"/>\r\n",
       "      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"104.589844\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"165.869141\" xlink:href=\"#DejaVuSans-64\"/>\r\n",
       "      <use x=\"229.345703\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "      <use x=\"261.132812\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"322.314453\" xlink:href=\"#DejaVuSans-66\"/>\r\n",
       "      <use x=\"357.519531\" xlink:href=\"#DejaVuSans-20\"/>\r\n",
       "      <use x=\"389.306641\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"428.515625\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"489.794922\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "      <use x=\"553.173828\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_31\">\r\n",
       "    <path clip-path=\"url(#pd419019c3e)\" d=\"M 56.463068 136.922727 \r\n",
       "L 117.080546 136.811673 \r\n",
       "L 126.651726 136.55432 \r\n",
       "L 131.437317 136.252275 \r\n",
       "L 136.222907 135.703828 \r\n",
       "L 139.413301 135.108775 \r\n",
       "L 142.603694 134.226383 \r\n",
       "L 144.198891 133.637419 \r\n",
       "L 145.794088 132.921908 \r\n",
       "L 147.389284 132.053698 \r\n",
       "L 148.984481 131.001819 \r\n",
       "L 150.579678 129.729613 \r\n",
       "L 152.174875 128.194236 \r\n",
       "L 153.770073 126.346115 \r\n",
       "L 155.365272 124.128717 \r\n",
       "L 156.960471 121.478478 \r\n",
       "L 158.55567 118.325992 \r\n",
       "L 160.150868 114.597474 \r\n",
       "L 161.746067 110.218335 \r\n",
       "L 163.341266 105.118323 \r\n",
       "L 164.936464 99.239235 \r\n",
       "L 166.531663 92.54545 \r\n",
       "L 168.126862 85.03725 \r\n",
       "L 169.72206 76.766777 \r\n",
       "L 172.912458 58.504053 \r\n",
       "L 176.102855 39.761167 \r\n",
       "L 177.698053 31.212877 \r\n",
       "L 179.293251 23.862072 \r\n",
       "L 180.88845 18.190483 \r\n",
       "L 182.483648 14.60467 \r\n",
       "L 184.078846 13.377273 \r\n",
       "L 185.674045 14.604405 \r\n",
       "L 187.269243 18.189983 \r\n",
       "L 188.864441 23.861358 \r\n",
       "L 190.459639 31.212008 \r\n",
       "L 192.054838 39.760202 \r\n",
       "L 198.435632 76.765849 \r\n",
       "L 200.03083 85.036396 \r\n",
       "L 201.626028 92.544662 \r\n",
       "L 203.221227 99.238557 \r\n",
       "L 204.816426 105.117749 \r\n",
       "L 206.411624 110.217812 \r\n",
       "L 208.006823 114.597032 \r\n",
       "L 209.602022 118.325609 \r\n",
       "L 211.19722 121.478161 \r\n",
       "L 212.792419 124.128423 \r\n",
       "L 214.387618 126.345894 \r\n",
       "L 215.982816 128.194052 \r\n",
       "L 217.578015 129.729436 \r\n",
       "L 219.173212 131.001679 \r\n",
       "L 220.768408 132.053617 \r\n",
       "L 222.363605 132.921834 \r\n",
       "L 223.958802 133.637316 \r\n",
       "L 227.149196 134.710742 \r\n",
       "L 230.339589 135.43554 \r\n",
       "L 233.529983 135.923868 \r\n",
       "L 238.315573 136.373536 \r\n",
       "L 244.69636 136.675676 \r\n",
       "L 254.267545 136.848264 \r\n",
       "L 275.005103 136.917175 \r\n",
       "L 310.099432 136.922668 \r\n",
       "L 310.099432 136.922668 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 43.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 322.78125 143.1 \r\n",
       "L 322.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 43.78125 143.1 \r\n",
       "L 322.78125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 43.78125 7.2 \r\n",
       "L 322.78125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pd419019c3e\">\r\n",
       "   <rect height=\"135.9\" width=\"279\" x=\"43.78125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 360x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 清除以前的梯度。\n",
    "x.clear_gradient()\n",
    "y.backward(paddle.ones_like(x),retain_graph=True)\n",
    "d2l.plot(x.detach().numpy(), x.grad.numpy(), 'x', 'grad of tanh', figsize=(5, 2.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 28
   },
   "source": [
    "总结一下，我们现在知道如何结合非线性函数来构建具有更强表达能力的多层神经网络结构。顺便说一句，你的知识已经让你掌握了一个类似于1990年左右深度学习从业者的工具。在某些方面，你比在20世纪90年代工作的任何人都有优势，因为你可以利用功能强大的开源深度学习框架，只需几行代码就可以快速构建模型。在以前，训练这些网络需要研究人员编写数千行的C或Fortran代码。\n",
    "\n",
    "## 小结\n",
    "\n",
    "* 多层感知机在输出层和输入层之间增加一个或多个全连接的隐藏层，并通过激活函数转换隐藏层的输出。\n",
    "* 常用的激活函数包括ReLU函数、sigmoid函数和tanh函数。\n",
    "\n",
    "## 练习\n",
    "\n",
    "1. 计算pReLU激活函数的导数。\n",
    "1. 证明一个仅使用ReLU（或pReLU）的多层感知机构造了一个连续的分段线性函数。\n",
    "1. 证明$\\operatorname{tanh}(x) + 1 = 2 \\operatorname{sigmoid}(2x)$。\n",
    "1. 假设我们有一个非线性单元，将它一次应用于一个小批量的数据。你认为这会导致什么样的问题？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 30,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/1796)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pte",
   "language": "python",
   "name": "pte"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
